An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews

Citation screening is an essential and time-consuming step of the systematic literature review process in medicine. Multiple previous studies have proposed various automation techniques to assist manual annotators this tedious task. The most widely used measure for evaluation automated citation Work Saved over Sampling (WSS). In work, we analyse examine its drawbacks. We subsequently propose normalise WSS which enables performance comparisons across different reviews. analytically show that normalised equivalent True Negative Rate (TNR). Finally, provide benchmark scores fifteen datasets with [email protected]% recall compare Precision AUC.